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ASSTRACT 

The Use of Multiple Strategies 1n Evaluating 
An Experience-Based Career Education Program 

Thomas Owens, Joseph Haann and 
Harry FehrenbaGher 

Northwest Regional Educational Lahoratory 

This paper dascrib^is the rationale for using multiple evaluation 
strategies in a Gomprahensive program avaluation, EKamples are 
given of eleven strategies ueed in the evaluation of the Experience- 
Based Career Education project developed by the Northwest Regional 
Educational Laboratory through sponsorship of the National Institute 
of Education. Strengths and limitations are cited for the use of 
pre- and posttesting in an experdjnental design, student followup and 
longitudinal studies, student case studies, content analysis/ adversary 
hearing f cost study, ethnographic study ^ use of local study coiranittee/ 
organizational study, panel review by ift^perts, and survey 
questionnaires. Six criteria, proposed for consideration in 
determining the best evaluation strategies to use, are discussed. 
These are cost, timing, credibility of findings, degree of 
obtrusiveness , amount of coordination needed, and efficiency. 
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THE USE OF MULTIPLE STRATEGIES IN EVALUATING 
AN EXPERIENCE-BASED CAREER EDUCATION PROG^l 



within the past two years i tha concept of educatianal evaluation 
has iDecome incraasingly assoaiated with that of providing and using 
inforsiation to analyze alternatives and improve the deGision-making 
process (Stuff lebeam, 1971), Much of the methodology of evaluation, 
however, has not kept pace with the needs for improved decision inaking. 
Frequently evaluators tend to think of using only statistically-baaed 
experiinental design Mdels^ perhaps because these have been emphasised 
in their own graduate training programs* Quite often a gap exists 
between the na^ow empirical findings of the usual evaluation report 
and the much broader-based factual information required for programmatic 
decision making. Traditional evaluation designs tend to severely limit 
the variables being considered, overlooking many other real issues. Such 
evaluation often emphasiEes the easily quantified elements while 
negleeting the more subjective, less tangible points that may be more 
directly related to the decision-making proeeas itself. For example, 
the decision whether to adopt a new schoel curriculum is seldom based 
solely upon the research data related t© students' academic perforitonce. 
Otter variables, such as the congruence of the new curriculmn with the 
existing school curriculum and facilities, the cost and willingness of 
the school board to fund the new curriculmi and the estimated level of 
acceptance of the project by students, educators and the conmiunity must 
also be considered. These t^^s of considerations are typically 
difficult or in^ssible to exmiine with the prevalent statistical 
models* Other suitable designs are needed to eKpand the evaluators* 
"bag of tools , " thus allowing then to select the strategies or condsination 
of strategies that best fit a given situation. 

Robert Stake (1975) has written atout his strong preference for 
&n alternative to traditional preordinate evaluation approaches, 
Preordinate evaluation, advocated by researchers like Popham (1969), 
relies heavily upon the prespecification of specific program objectives. 
Stake states that preordinate plans for the evaluation of educational 



programs en^haslze " 1) statement of goals, 2) use of objective tests, 
3) standards held by program personnel, and 4) researoh-type reports" 
(Stake, 1976, p. 10) . The alternative favored by Stake is responsive 
evaluation. "An educational evaluation is res^nsive evaluation D 
if it orients more directly to program activities than to program 
intents, 2) if it responds to audience requirements for information, 

and 3) if the different value perspectives of the people at hand are 
referred to in reporting the success and failure of the program" 

( Ibid . , p. 10). 

The authors of this paper argue that in a comprehensive evaluation 
that alms to sarve the information needs of multiple audiences there 
is a need for both a preordinata and a responsive evaluation. Legitimat 
audience needs generally incluae answers to both the question of how 
well a program achieved its stated objectives as well as what program 
participants actually did in the program and why. 

After presenting a brinf description of Ijcperience-Based Career 
Education (EBCE) and Its demonstration project in Tlgard, Oregon, callei 
Conmunity Es^eriences for Career Education, CCE)2/ this paper will 
discuss eleven evaluation strategiea that have been used with (CE)2i 
assesH the strengths and limitations of each atrategy, show some 
relationships among the strategies, disouBs ways of cornmunicating 
evaluation findings, and present some suggestions useful co others 
conteinplating the use of multiple evaluation strategies. 



n. COMMUNITY EXPERIENCES FOR CAREER E0UCATI0N--(CE)2 

community Experiences for Carear Education^ (CB)2, is one of four 
Experience-Based Career Education (EBCE) programa being tested under 
the auspices of the Natidnfil Institute of Baucation. Qjerated in 
Tigard, Oregon by a nonprofit eomnimity corporation, the program is 
directed by the Northwast Regional Educational Laboratory (NWBEL) . 
(CE) 2 is composad of approximately 60 high school juniors and seniors 
and provides a conprehenaive high school education through experiences 
in the connnunity. 
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A primary geal of the (CE)^ program has bean to intsgrate a 
student's knowledge of a variety of careers with the acquisition of 
cognitive^ interpersonal and affective skills through a series of 
planned es^ariances with identifiad laarning outcc^as. anphaais is 
placed on the assumption of individual student rasponsibility for his 
or har own learning. Four Gharaeteristica help describe the essential 
elements of (CE)2' 

1. The learning program evolves froTO adult activities in the cosanunity. 
It is reasoned that if the learning activities are based directly 
on adult tasks and roles in the coimnunityp laarning will ba 
recognised as more relevant by youth in preparing for the ■ 
transition to adulthood. 

2, The program is based on experiential learning, actively involving 
students in the dally work of coimunity life* TOis "hands^n" 
approaeh to laarning has long been recognised as an effective 
learning strategy and (CE)2 i^ attempting to implemant this 
approach in a comprahensiva program* 

3. The curriculum of CCT)2 is fully integrated. Just as the salesman 
or forCTian does not think of his interactions with people strictly 
irr terms of granmtar, vocabulary or psychologyj the (GEjj curriculum 
also applies no artificial distinction between the "disciplines * " 

4, (CE) 2 is a fully individualized program. The learning goals as 
well ai the learning strategies are varied to meet the needs, 
interests and abilities of each student (toens and Fahrenbacher ^ 
1975), 

Although the EBCE deTOnstration aits operating in Tigard# toagon^ 

differs from the EBCE versions developed at the other regional educational 

laboratories in areas such as program governance, it contains essential 

2 

characteristics comron to all four versions. 

imring the 1975-^76 school year* the EBCE model developed by 
NTOSL was successfully implemented in four pilot sites in the Pacific 
Northwest , Due largely to funding made available under Part D of the 
Vocational Education Act by the U*S. Office of Education r the EBCE 
program developed by the four regional educational laboratories will 
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becoma operatior il in new sites in 42 statas dicing the 1976*77 scSool 
year* Since soine of the individual eleffients of EBCE are shared by other 
comprehensive sedondary education progrMis* the evaluation challenges 
presented by EBCE can be. applicable to many settings today. 

III. EVALUATION STRATEGIES USED WITH (CE)2 

A conprehensive forinative and siananative evaluation of the (CE)2 
prograffi has occurred over the past three years that looks botii at 
outcomes and what Cronbach (1975) terms "mediating events." This 
evaluationr while focusing on student learning, 1ms also examined 
other areas such as program management^ costs and employer and 
community involvement. 

Because the (CE) 2 program was designed to achieve a wide range 
of cognitive and affective outcomes in an individualized manner , the 
evaluators realized that no single research methodology would be 
adequate. Tlierefore, a combination of various methodologies was 
employed under the assumption that the weaknesses of any one method 
would be counterbalanced by the strengths of another* 

In planning the (CE)2 evaluation activities for 1974-75 a detailed 
evaluation design was prepared by the mBEh evaluation team that include 
a matrix of (CI) 2 outcome goals along one dimension and evaluation 
instruments along the other dimension. TliuSr for each goal, at least 
one primary and one secondary instrument were identified that would 
provide information addressing that goal. Since there are other types 
of evaluation questions asked by various audiences that go beyond the 
project goals, a second matrix was developed in retrospect. This 
second matrix involves an evaluation design in which some of the broader 
important evaluation questions are displayed along one dimension and 
separate evaluation strategies (each focusing on separate issues and 
audiences and containing its own set of instriutients) are displayed 
along the other dimension. 
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Each of the alaven strategies involve^ tiie devalopment and/or 
use of different evaluation instr»aments. Table 1 on the following 
page shows a sample of evaluation questions that require multiple 
evaluation strategies. As indicated in that i table, some questions 
can be answered through more than a single evaluation strategy* 
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Table 2 displays the evaluation strategies by intended audAenees, 
Five baeic audiences eeem applicable to EBCE and other large-scale 
developnent projects: 1) the funding, agency it in this case, NIE) , 
2) the research coranmity, 3) persons in school districts considering 
the potential adoption of the project^ 4) parents and local conmunity 
where a project is being developed, and 5) the project staff. As with 
Table li each evaluation strategy can answer more than a single 
question and can also serve more than one intended audience. Although 
the reader may disagree with the authors' selection of prijnary and 
secondary audiences for any particular strategy, several points emerge 
from Table 2. Information resulting from some strategies is probably 
of little or no interest to some audiences. For exanple, results of 
a local district study conmittee are probably of no Interest to the 
research community while the comparativa testing occurring with a 
true experimental desi^pi framework is likely to be of high interest 
to this group. A second point illustrated by Table 2 is that some 
audiences I because of the nature of the decisions they need to make, 
will be interested in information resulting from a ^eat variety of 
evaluation strategies* For example , educators in districts that are 
considering the potential adoption of thm IBCE project are likely to 
have a much greater variety of information needs . than are parents and 
members of the local community in which the project has been developed 
and operated. 
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ANALYSIS OF EViiLUATION ST^TEGIES BY INTENDID AUDIENCES 



Type of Evaluatlen Strategies 

Student Outcome Focus 

1. Conqparative testing with 
eKperimantal design 

2. Student followup or 
longitudinal studies 

3* Student case studies 

4- Content ^^lysis 

Program Outeoroa Foous 

5* Adversary hearing 

Cost studias 

Ethnographio study 



Intended Audiences 



6. 
7. 
8. 



9. 
10. 



Local study coiraiittea 

Orgmi nation al study 
Panel review by es^erta 



NIE 



S 



s 
s 
s 



11 » Survey questionnaires 



P 
P 
p 



Hesear^ 
Conrounity 



S 



Adoption 
Sites 



P 
P 
P 

:P 
P 
P 

s 
P 

p 
p 



Parents 
a Local 
^nffli^ity 



P 

S 

a 



Project 
Staff 



S 

s 



s 
p 



* P represents priinary audiences for each evaluation strategy 
S represents seconda^ audiences for each evaluation ^trate^ 
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Each of the alaven avaluation strategies identifiad in Tables 1" 
and 2 is described briefly below. Each strategy has a partioular 
audience and sat of avaluation instrijments from which data wera collected 
and analyzad. 
1* CajngM^ativa Testing 

In order to detarmina student gains in Basic Skills, Life Skills 
and Car ear pavelo^ent which are attributable to participation in 
EBCEf an axperijnantal design was used in 1974-75 that involved 
pra^ and post testing of the experimantal and control group 
students* From the total pool of student applicants for (CE)^* 
students were randomly assigned to participate in (CE)2 or the 
control group, students in the control group ramained in the 
ragular school program throughout the year. Instruments used 
included standardised tests^such as obtests from tha Comprehensiva 
Test of Basic Skills^ instruments developed by Educational Testing 
Service that ware used commonly on EBCE programs in three other 
parts of the country, and instruments developed locally by NV^EL, 
Correlation and psychometric analyses were run to detamina the 
relationship among student outcome measures and between various 
outcome and student background measures, \ Thmnm analyses ware 
also run jbo^assess the reliability and validity of the instruments 
usad and to datamine the differential progrMi affects upon various 
(CE) 2 students , 
2. Follpymp Longitudinal Studies 

One of the tasks of the Educational Testing Service in its 
^ externar evaluation of ^CE during 1974-75 included the followup 
of students who have graduated from (CE)2 since it began and of 
students who had dropped out of tha program. Special student 
interviews ware davelopad and used for this purpose, ^e intent 
of this study was to assess the effects of (CE)2 upon studants' 
occupational, educational and personal life after they had graduated 
from the program, A study was conducted the following year to 
assess tha first and second year ijnpact of {CE)2 on students who 
rKnained with the program for both years. Preliminary thought 
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was alkso given to a longitudinal study of the (CE)2 9^^^^^*®^' 
although the nianber of such graduates at the time was rathesr : 
s mai . * ' 

• • • " . ' i 

Studmt Case ptudieg ' 
Intansiveroase studies of six (CI) 2 students saleeted by ^ \ 
stratifi^ random san^le were conducted by tb.e NWREL evaluation J 
team during 1974-75 , Because the (CE) 2 program is compleK 
highly individualized and involves multiple, interrelated 
learning strategies and a wide spectrum of student outcomes, 1 
the case- study approach to evaluation seemed appropriate, it 
is in keeping with the philosop^ and praatice of (01)2 and ; 
concentrates on the individual student. Case study students t- 
were selected so as to represent juniors and seniors and \ 
students with high and low motlvationV Multiple data collection \ 
strategies we^re used in the case etudiee to obtain a cross 
validation of information about each of the students ^ These 
methods included I a) observation of students at ^ployer sites j 
three tunes a yaar and interviews with the students T employer 1 
instructors # b) parent interviews, c). indepth student interviews 
four ttoes during the year, d) interviews with progrMi staff, , 
and ej a review of student-coinpleted projects and other dociments* 
h total of 23 records were identified as secondary sources of 
data for each student and a set of guide questions was prepared 
for analyzing each secondary soiurca* A irore complete description 
of tha rationale, procedures and findings from the case studies 
is avail^la {Fehranbacheri Oweng and Haenn, 1976) . 
Content Analysis ? 
Since EBCE is a highly individuallEad program and its oparations . 
and products well docmnented, tha avaluators agreed to usa student 
and project documents as an mportant part of the evaluation* 
Content analysis (the systematic classification and use of existing 
documentation) became a valuable tool for transforming existing 
file data into a form usable for evaluation. Conttot analysis 
was applied to four types of datai 1) student projects and ^ 



written reports, 2) student records of program activities begun 
and completed, 3) the list of employer sites used with (CE)2# 
and 4) the (aOa Board minutes. 

The Life Skills projeets and resulting written reports for each 
of\^the six case study students were retained by the learning 
managers throughout the year and given to the MWRIL evaluation 
unit for analysis, Thm evaluators identified 12 criteria they 
wished to apply to each student project* ^ese criteria included 
the extent to which Basic Skills work was integrated into the 
project, the mxtmnt to which the project fit a student's int&rest 
areas ^ and the extent to which the project met toe objectives 
for the Lif^ Skills area in which it was witten. A two-page 
written description of each student was prepared together with 
a rating guide involving a five-point scale for each criterion. 
An experienced high school e^riculwi director not familiar with 
EBCE was hired as a consultant and spent four days applying the 
criteria to each case-study student's projects • ^e consultant's 
ratings were keypunched and analyzed for descriptive statistics 
and an assessment of the general areas of strengths and weaknesses 
of projects in each of the five Life Skills areas was reported. 

Student records of program activities begun and completed were 
recorded by the project staff and provided to the evaluators 
for analysis, !I5iese data allow toe evaluators to learn the 
average length of ttoe needed to complete projects and other 
- activities^ the coc^etencies most and least f recently completed 
and, on a time trend basis # toi deterinine toe pattern of progrm 
activity completion over the co^se of the entire school year. 

The list of cooperating employer sites was analyzed and 
categorized into the 15 U.S. Office of Education occupational 
clusters in. order to see which clusters were most and least 
heavily represented and to determine if r.-y clusters were 
not represented. 

The monthly {CE)2 Board minutes were analyzed to determine the 
content nature (and frequency) of. „ the Board's discussions and. 



to determine whatoer most of thm time was spent by the Board in 
listening to progress reports, discussing operations or 
discussing policy raQommendations • The results of this analysis 
were displayed by misnth and indiaated that the Board was, in : 
fact, a policy making Board, 

5p Mver^ary Hearing . ' . \ 

As a way of providing infomation directly relevant to pcftential 
adopters of BBCE, a prototype adversary hearing was held and 
videotaped. Thm hearing, based on issues considered relevant 
\o the adoption decision by school district administrators, 
presented witnesses and argtanents favorable and unfavorable 
to the adoption of EBCE and provided for limited cross eKamination 
of witnesses and testimony presented. Two consultants from 
Midwestern universities served as the adversaries, h professor 
of law and several other consultants assisted the NWML evaluators 
in planning and staging this abbreviated hearing* Alttough the 
prototype hearing was quite limited in time and resources utilised, 
the general approach seems to have merit in future evaluations 
(Hiscox and Owens, 1975), 

6. Cost Studies 

, Two cost studies regarding (CE)^ operations were conduoted by two 
outside consulting groups* One study investigated the ongoing 
costs of {CE)2 in comparison with secondary academic and vocational 
progrMis in three selected Oregon school districts. Direct daily 
staff costs of instruction for individually sampled students were 
obtained but lesser costs such as physicai plant, e^ipment and 
transportation were not computed* By computing direct instructional 
costs on a sampled student basis, it was possible to exanine the 
wide variation of student costs within the same academic program, 
as well as to compare progrm totals (Gourley, Gourley and Delos, 
1975). 

In the second cost study conducted in 1974, an investigation was 
made of the costs being absorbed by employers when they accept 
a (CE)2 student or students on assignment at thei r sites 
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(Institute for Educational Management, 1974) . Costs were 
analysed in terms ofi apace oeGupied by the student .equipment 
used in instruetiQn, administrative costs # aonsunable supplies 
related te instruetion and instructional tijoae* , A representative 
aan^le of 30 out of 95 employer sites was selaoted based on 
stratification^ size and t^^ bu^inese, aaployer interview 
data were analysed separately for students on an e^loratory 
level and those on a more intensive learning level. 
Etlinpgraphig_ study ^ 

One of the strategies used by Educational Testing Service as 

part of its EICE eKternal evaluation contract with HIE was ttiat 

of employing anthropologists to conduct an ethnographic study 

at each of the four EBCT demonstration sites. 

study of Cai)2 was intended to hm descriptive rather than 

evaluative, "They are to provide a tackground for toe 

interpretation of syst^atically gattered evaluative data ^ 

rather than serving as a substitute for the latter," (Durgin, . 

1975, p* 3). ^e resident anttropologist; spent approkljnately 

-t ■ ■ . ' . ■- - - . ■ ■ ■ . = 

seven TOnths in daily contact witii CCE)2 to describe the informal 

and formal activities Uiat characteriEe the project's; learning 

center and employer sites* The main focus was "on student 

behaviors and conceptions rather than on those of staffs 

employers or parents*" (Ibid, , p, 4) , /Methodologies used 

included student obser^ . 

students and staff. > 

Local Study CoBroittee Review 

In 1974 the Tigard School Disteict f oMted a study committee for 
six months to make reconQnendations regarding the future of (CE) 2 * 

nr»h j g 1 ^^TTiCTbey jjoTmni tt m vim eontpo sed of the (CE) 2 Dl^eector , 

several p^ticipating employers # two parents v a student and 
administrators from the Tigard High 8ch<»l and tiie Die trio t* 
TOie^ eonoaittee reviewed , national literatyre related to educational 
alternatives ^ studied tte NWffiL interim evmluation findings, 
and conducted a questionnaire suwey of participating CTiployers 
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■ " ■■ . ' ' .... f 

$ ' ' ■. . ■ i 

.... . . . . , ^ 

to determin© tiieir continued corafiittnent to CCE)2 providing the 
Tigard iohool DietriGt assianed responsibility for funding and = 
operating Uie program (Beiar^ at, al## 1974 =^ 1 

9* Organizational Stud^ \ 
M p ar t of the external smtmativa svaluation Gon tract. Educational j 
. Teating Serviee condUGted an organiEational stu% of raCE at eaoh j 
of the four demonstration sites* A sociologist was responaibla j 
for this assigraient. The sociological analysis foeusad upon the : 
progrOTi's history , its ©rgani^ational structure and | 
interorganiEational relationships CWask^^^^i^ for j 

this report .were based upon interviewe ^ith^^^ 

each of the four regional laboratoriaa participating^;^ j 
development of EBCE and on aMlysls of institutional records, | 
particul^ly the quarterly progress reports submitted ty each 
Laboratory to NIE. ■ ] 

10. Panel Revlaw by Ibgperts ^x .-^^ 

During the first and second years of IBCE operation, agpanelr^.^" ; 
review tCOTi was used. In tiie first year the team that visited 
and reviewed {CE)2 consisted of a college dean^ who is nationally 
knora in career eduoat ion # and the personnel manager at one of 
the large p^ticlpating enployer sites in Oregon* Their comments j 
helped influence revisions in. the program for; the f^ ; 
year* In toe second year of opffl-ation, NIE coimisaioned a five^ ; 
person site review team made of. an assistant, superintendent 
for currlculiam and instruction of a Midwestern school distelct, 
an education progrMi director of an International corporation, ; 
a Teamsters Union director of research and education, an j 
executive of a private research institute ^and the dean of a _ \ 

college of education, ^is tem ^ent three days at the f 
dmonstration sites for each of the four participating educational 
laboratories obtaining information on evaluation questions or 
issues related to toe project's objectives toat were agreed on 
jointly by the project staffs at the four sites, Nil Md the 
review team menters. Prior to visiting each site, toe team 
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reviewed each site's operation plan, ^arterly re^rts and 
evaluation re^rts* Upon arriving at each site, the team 
received a comprehejisive briefing about the obj actives and 
activities at the site followed by obaervation'Sf s at 
tiie Iiearning Center and at employer sites and interviews with 
a sample of students, parents, employers and staff , Ximnediate 
feetoaok was given by the team to the project staff at each site 
at the conclusion of the visit. This was followed by a written 
report containing conclusions and recommendations, 
11 f Survey^ Quest ionnair eg . . 

The evaluation team has made eKtensive use of questionnaires to 
survey perceptions of the {CE)2 program from various populations 
including students, graduates # staff, parents arid employers* 
Questionnaires have also been used to elicit self ^report data 
on student progress. c - . ■ 

The primary audiences of questionnaire survey f indi gs have been 
the developmental staff, NIE, parents and local coamiunity, and 
potential replication sites . ^is type of information also may 
be useful to participating school districts. ^; ^ 

During the first year of evaluation questionnaires were utilized 
to obtain employer appraisal of overall student performance and . 
both parent and student appraisal of the (CE) 2 pr pgr am. In the , 
second year, students, parents and employer Sj'resp^^ 
Slaveys, rating formtf A T(CE>^ ;yisitbr 7 . 

questionnaire was also used, as .was a project director 
questionnaire which served as a common base for describing 
essential characteristics of all four sites that were of interest 
to NIE. In this second year most of the study ins trmnents used^ 
w^e developed cooperatively by evaluation staff members from 
the four regional educational laboratories and from toe 
Education and Work Group of KIE, These instruments contained 
a coiranon section relevant to all four sites and a unique 
section that ^ allowed each site to measiire variables of local 
interest. This combinatlon'^thus^llowed^or^com 
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naade of NIB across sites as well as the speGial naade of ^aqh 
site* 

Most survey questionnaires used in the 1974^75 evaluation, of 
EICE were developed jointly by Educational Testing Service and 
the internal evaluation staff at the four regional educational 
laboratories* TOese instruments were administered to pro j eat 
students, staffs parents, miployerSr project graduates and 
control group students* 

IV* STRENGTHS AND LIMITATIONS OF EACH STRATEGY 

Examination of the multiple evaluation strategies used with 
(CE)2 uncovered certain strengths and limitations in each strategy. 
Table 3 identifies the essential characteristics of each strategy as 
used with (CE)2 and the authors' assessment of perceived strengths 
- and limitations of each,^ This assessment should be useful to evaluators 
considering the use of such strategies in future evaluations , 
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.'Evaluation strategy 

STUDENT OUTCOME FOCUS 

C^paritivi testing 
within an experiniental 
desip 



AiSlS8f®T OF EVALUATIOT STEraGllS UffiD 



Bigantid. Characteristics .. ' Strengths 



Comparei project studints' 
perfsrrancs and progreis witt 
that of control and comparison 
iroup students to ditennins 
the prograni's trsattiint affects 



1. Attiipts to control 
■"■""factors jiopardizinf / 

internal and external 
validity 

2, Genirally accept^li 
standardi and procadures 
for judging signifiGint 
differencas betwean 
groups 



Graduate foUowup 
lo,ngitudinal study 



Assesses tiie prograin's tffscts 
upon students' occupationil, 
educational and perionil life 
after tliey graduate from the 
program 



Long-range sljudent 
outcoinei of tiii prggrain 
are important and can 
only be assessed after' ^ 
students have graduated' 




Limitations 



1. Ignores variations 
existing within ' 
treatmsnts ■ 



2 r Can interfari with m B 
natural Qperations of 
tiii prograiti ; ;• 

3, Often limited to only ; 
a few generalised outcoiiiv 
measuris,' • ' 

- ' - - - — ^ 4- 

Ai evaluitlpn gits ; ; 
inori reiTOta In tini ■ ■ ■ 
' f OT the pfograi / ^v: 
trsatifisnt/ it bioonis - 
mm difficult -to 
istibliih Giuiility ' . 



Studsnt om itudies 



Provide evaluatien and indaptt 
^ discription of. a iiipli of 

studinti.- pirfomanciij 
Ittitudas and intiriations 
witJi pairs, paKntii itaff and 
amployari 



L Donifflint individuil 
traatoint of studints 

2. Allow for a iynthaiii of 
ittuoh data ^ut 
individual: itudinti 

3. . utilize raaliitle : 
subjeetivi jiidpant for 
interpratation 



L Uii of fiw studants. 
makss ginirilizations , 
to thi total population 
difficult. 

2, Difficult to datfOt > 
thi ivaluatori- 
potential biaiis .. . - . 



Evaluation Strategy Essential Charaeterlstics 

4, Content kalysis Convsrti agisting pregrai 

doeuinantation into usabla 
fom for ivaluation purpossi 



Strenqths 



S, Mvirsary hiaring 



6, Coat Studies 
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1. Presents oppoiing argiients 
and witnesasi favorabla ad 
unfavorable to tiie prograni 

2. Provides for cross 
examination of witnesses M 
testiiaony rilated to the 
potential adoptim of the 
prograra by other districts 



1, Use is mide of 
availabii progrffl 
docuinentation 

2, Un^trusive lethed 

3, Mi coUsctid an 
usually diractly 
relevant to the program 
operationi 



Lisiitations 



1. Dependent upon the 
accuracy of the project 
staff in cgllicting and 
ricording the inforEiation 

2, Missing data may be 
impossible to retrievi 
or istuiate 



1. Presents both pro Md : 


li Decision lakers niiy 


con evidence i 


, be influincid by the 


h Provides for a cross 


persuasiveniss of the 
adversaries more tiim 


exaiination of testiniony 


by their evidenei 


3. Particularly attentive : 


L The qualifications of 


to the infomation 


the two adversariis 


needs and time frma 
of decision inakers 


lay not te balaiced 


considiring program ! 
adoption 


1. Time limitations may 
cause a foeus on oily 



a fiw iisuai 



h Provida information about 

the diract md indirect 
' . costi fer employars 
i participiting with the 
program 

2. Conipara the program* s cost 
with coaipating prograni 



L Diraot daily stiff L Staff CQiti art 
. CQSti of instruetion for computsd butothir 
individually saffiplsd cQStir iuch as phyiieal 

itudints ire Qbtained plint, equipMt aid 

transportatien may not 

2, Dsvalopfflintal and bi 
instructional aosts of 

the program are isolatid 2* No attempt usually is 

made at a cost/binafit 

3. ^poftunity costs to itudy 
imployirs ara assissed 



jyaluatiQn gtratagy 



Essential Charasteristies 



Straigthi 



Liniitatieni 



^7, ittnograpliic Study 



1. Deserites the behaviors of 
progratB students in depth 
snd depiets thiir intsr- 
actioiii with pan, staff 
and aiployars 

2. Diseribas thi fonnal and 
infomal structura of the 



li Loeil study 
comitteg revia^s 



Kgviaws ixiiting evaltjition 
data and intsgratai them 
witii a new survey in order 
to idantify altirnative ' 
rQComnendationi rigarding 
tti future of the program 



1, Intansive ^scriptiai 
of the prograa based on 
daily ^isrvation 

^" intiraction of tia « 
anthropologist 

2. Ginirally mohtrusive 



1p ReaiiiSiis the naad for 
till progrra . v 

2, Evaluitas each propssed 
altirna.ti^ 

3, Actively involvas aeine 
parints md eoranunity 
in aviluatign 



1. Not easily si^jiot to 
raplicatiai 

2, Eiquiras training and 
talent not available to 
BiOit evaluition taaii 

3* liplicit value judgraents 
of the anthropologist ' 
are soBitiniis hard 
to diteQt 



1. Such coaittiis of tan 
laak nioasiary skiil 
in intirpreting 
avaiuation findings = 
^or designing niw 



instrifflients 



9, Organizational 
study 



i Integratas availabla 
aid newly collietid data 



Invastigitii the L Applies iound 

organizatimal structuri and iodQlogiQil constructs 
diVilopniant of the program and to tHa study 
its intirofganiiational 

relationships 2. IntigratiS manigamant 

reports and pirsonal 
intarviaws 



1, Focusis upwi a narrow 
iipect of the projaet 

2. ffle audisnca for this 
study is more limited 
than that for ether 
avaiuation studiis 



5 er|c 



0 



0 



iviluation Strategy Egsantial Charactjristici 



Strengths 



10; P^iil review by 
ixpirts 



Hi Surviy questionnaires 
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Pavisws existkg nianagaisent m& 
avaluaticn raporti, involvas 
intinsive onsite observations 
and interviews followed by an - 
eril dabriifing aid latir 
writtan report including 
Goncluiiaii mi recomndations 



Obtain percipticns of the 
prograffl froni students. 
Staff? pirints and employers 
and obtain self report data 
on studant progress 



L Allows external esprts 
to apply a fresh 
prspioti^ in 
intaractinf with 
prograBt'reiated people 
and in caching 
cenclusiens md 
reconfindations 

2. Allows exparti from 
various fieldr to work 
as ,a tean in their 
projaet reviiw 

3i Allows for the m of 
natioially knowi 
talents that eould 
not be affor^d m a 
fuUtiii baiis 



L Can obtain peopli's 
opinions in a 
confidential manner 



li EKtirnal raviiWiri 
soBetiBiei fail to 
CMttprehind the 
intsntions or rationale 
for tiie prograi-s 
oparaticm 

2* B^firts soBitinies let 
ttoir parsenal values 
intirfere with their 
. regomendations 

3» Mcaurati iipressions 
are songtines ac 
bioiuse of the 
brevity of tine 



Sabjict to fespondent 
iniiinterpretation or 
bias 



2, Ginerally economcal to 2. May deal superfidally 
collect with issues 



31 



ERIC 



V, RELATIONSHIPS AMONG THE EVALUATION STRATEGIES 



The ralationships Mftong the alaveii evaluation strategies described 
in the previous section are complaK. The selection of these strategies 
was not done in advance thrQi^gh some grand design but rather evolved 
out of the expressed needs of the various audienees identified earlier 
in Table 2, Nevertheless, certain patterns emerge in retrospect that 
may be useful in designing future large-^scale project evaluations. 

As was argued in the introduction of this paper, a comprehensive 
evaluation needs both preordinate and responsive evaluation strategies. 
When viewed from this framework, the comparative student testing with an 
experimental design setting and the student followup/longitudinal 
study are primarily preordinate evaluation while the others are primarily 
responsive evaluation strategies. 

Pour evaluation strategies primarily were used to assess student 
outcomes s comparative testing, student followup studies^ student case 
studies, and content analysis* The student case studies allowed the 
evaluators to gain new insights into indirect outcomes of the program 
which resulted in the addition of several new areas to be included on 
the student end-of-year questionnaire and the graduate followup 
interviews. Content analysis of student projects and resulting reports 
were part of the case studies and provided insights for some questions 
asked of all students on a year end questionnaire. Descriptions from 
the case studies were useful also in helping to account for significant 
differences found in the experimental design study* 

The ethnographic study, while not focusing specifically on student 
outcomes, did describe student relationships with each other, the staff 
and^ployers* In areas where they overlapped, the ettmographic 
descriptions were generally consistent witii those of the case studies • 
They were often based on a single student, however , and covered a 
wider perspective of that student's environment than did the case 
studies. The ethnographic description also differed in that it reported 
more frequently upon serendipities in student behavior gleaned from 
the anthropQlogist's daily presence at the learning center for 
-approximately seven months. 



strategies to assess program outcomes as distinct from student 
outcomes were seLdom closely assuaiated^ ©Kcept that prior years' 
(CE)2 evaluation reports (involving foTOativa evaluation, comparative 
testing and survtiy questionnaire results) were reviewed and used by 
people involved :-n the adversary hearing i panel review meaObers and 
members of the local district study committea. 



VI, COMMUNICATING EVALUATION FINDINGS 

Each of the evaluation strategies performed by personnel outside 
of NWRBL, such as the ethnographic study by Educational Testing Service, 
has resulted in witten reports* The expert review panel has also 
made an oral presentation of its findings to NIE and to certain- members 
of Congress and their aides* The local study committee, in addition 
to its wit ten report, also made an oral presentation to tiie Tigard 
School Distcict Board of Education* In contrast with the written 
reports, the adversary hearing was videotaped and the edited tape 
served as the final produot* 

The 1974-7fB BBCE evaluation report prepared by NWREL was organized 
along three dimensions i an executive suimnary, findings organised around 
the project objectives, and findings organized by evaluation instruments, 
The executive sur^imary reported only the highlights of the report. Since 
some instruments covered more than a single project objective and since 
some objectives were measured by multiple instruments # it was felt that 
a second organizational structure was needed that listed each program 
objective end all the evaluation results related to that objective. 
This structure was developed especially for practitioners who were 
more interested in the project's outcomes than in how they were 
evaluated* The third structure, intended largely for a research 
audience, organized the findings according to the evaluation instrmnents 
used. This third approach included a description of each instrument 
and the populations with which it was used. 

Requests from various audiences resulted in the need for separate 
evaluation summary documents. One summary was intended for parents, 
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amployers and edneators wishing a layman's viaw of the avaluation. 
This led to the development of an evaluation digest of about six 
pages that covered the main findings. Other audienoe requests led 
to the preparation of a two-page evaluation sulfonate . Overhead 
transparencies that mmsmmsLzm eight or ten of the key evaluation 
__f in4 ings ha^a_al so_ various^-groupa*^- ^Th e— — 

EBCE evaluation findings eolleeted by all four regional aducational 
laboratories have been sOTimarized by the National Inatitute of Education 
and included in nationally distributed EBCE broohures. 

Perhaps the moat important and sueoessful strate^ for 
conmiimlcating the evaluation results to interested parties and to 
potential adopters of EBCE has been to insure that the EBCE staff 
members at each of the regional educational laboratories reaponaible 
for assisting school districts in the adoption of EBCE are fOTiiliar 
with the findings. These persons provide information and technical 
assistance to districts wishing to use EBCE, The evaluation findings 
are often appropriate to use in tiieir presentations and in responding 
to questions from viaitors or potential adopters. 

VII. CRITERIA FOR SELECTING MULTIPLE EVALUATION STRATEGIES 

T^e evaluation strategies described in this paper would probably 
be useful in many large-scale development projects where the need is 
to provide evaluation infoETnation to a variety of audiences. Because 
time and evaluation resomrces are important constraints , it is generally 
wise to consider a variety of potential evaluation strategies and then 
to analyse thm alternatives so as to select the minimal nun^er of 
J. strategies needed to accomplish the job. Factors important to consider 
in determining priorities among poitential strategies include i 1) costs, 
2) timing* 3) credibility .of findings, 4) degree of obtrusivenass, 
5) TOKjunt of coordination needed, and 6) efficiency* 
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Costs 

Thm eleven evaluation strategies descried in thii paper vaiy 
widely in oost. Thm gon^arative tasting, student ease studies, and 
ethnographic study were the most costly • The cost of some strategies 
such as the adversaj^ hearing and loaal study committee review depend 
heavily— on^contrdbuted—^ime-o£=nonproj eet=-per sonsr— ^ The^amoun t~o j=— — 
student and staff tuae involved should also be taken into consideration 
in deterndning costs. 

Timing 

The timing of an evaluation strate^is another crucial factor. 
Several elements of timing need to be Qonsidered including^ 1) the 
deadline when the Information is needed, 2) the lengto of time it 
would t^e to plan, collect and analyze data, and 3) the most appropriate 
tiiha in the developmental c^cle of a project for collecting certain 
data« 

Credlbnity 

Credibility of the findings is something that is often overlooked 
until an evaluation is completed and the data reported* Two exMiples 
from the (GE)2 evaluation can be cited where .the nature of the 
evaluators involved contributed to the credibility of the findings. 
In the adversary hearing, it was decided that the advocates for ajnd 
against the future adoption of EBCE should be completely independent 
of the project, Also, when the national panel review team appointed 
by NIE was created, its credibility was enhanced by having on toe 
team representatives of higher education, public schools f labor, 
industry and a private research organization* In additiori, credibility 
of the evaluation findings can be e^anced when several evaluation 
strategies produce results tihat reconfirm or support what was found 
through the use of othM strategies* 

Degree of Qbtruslvaness 

toother ijnportant factor to consider in selecting evaluation 
strategies is the extent to which a given strategy will be obtrusive 



Md p^haps clash with the mission and activities of the projact. ^ 
Concern for maintaining a "low profile" was one of the key factors 
in considering an ethnographic study wheirn a trained antoropologist 
coxild observe witl^ut^disturbing J^he enviroranant. ^e use of a true 
-= — — — — -e^i>eriinentai=-design^n"evalua^^ (CE)2 during the third year # because 
of its obtrusive nature , required delicate negotiation with the 
project operations staff* 

CQordi nation 

A point often overlooked in selecting multiple evaluation 
strategies is the amount and t^e of coordination needed. This 
coordination includes the Mioimt and t^e of interference tiiat may 
accrue to students, staff and others such as participating employers. 
It also includes the coordination of persons or agencies to be involved 
in the data collection for each strategy. One compromise the authors 
have used with the operations staff in managing the coordination of ' = AT 

different evaluation strategies was the agremient at the beginning 
of the school year as to the maximum total amount of direct time of 
students and staff that would be used in any=type of evaluation* This 
agreement then insured that the project's coordinator of research'^^d 
evaluation would maintain a close planning and monitoring of student 
and staff time used in svaluationp 

Efficiency * 

In serving as a discussant for an earlier draft of tois paper 
presented to the Fourth Annual Pacific Northwest Iducational Research 
and Evaluation Conference in May 1976 ^ Joseph Hansen of toe Portland 
Public Schools suggested an additional criterion* efficiency* be used 
in considering multiple evaluation strategies* He stated that "a 
strength of multiple strategies lies in the redundancy of information 
produced which provides thm basis for establishing the reliability of 
findings, ^is redundancy is costly and may be obtained at the 
es^anse of foregoing the collection of o'^er- bits of unique information,"^ 

^e six factors discussed above are not meant to discourage 
oth^ evaluators from using multiple evaluation strategies when appropriate, 
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Rather, they are memt to serve as a balance against the indiscrim^ 
inate use of a "shotgun" approach to evaluation. Am a result of 
oonsidering the evaluation strategies presented in this paper , it 
is -hoped- that other evaluators will give ttought to considering 
the xxsm of a wider "bag of trioks" and will select those that are 



niost appropriate to a program's needs. 
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FOOTNOTES 

\. 

^fhis paper is based on an earlier varaion presanted in Seattle # 
Washington at the Fourth Annual Pacific Northwest Educational 
Rese«oh^and_^aluation^ ConfareM 6 • ^ _ ^ ... 

2- - - _ 

An EBCE program has also bean davaloped, pilot tested and dissaminated 
by the Appalachia Educational Labor ato^, Ino,, in Charleston i Wast 
Virginia; Far Hast l^^ratory for Educational Rasaaroh and^^valo^ent 
in San Prancisoo^ California; and Raseareh for Better Schools i Ina* 
in Philadalphia, Pennaylvania, 

3 ■ . 

The authors wish to egress appreciation to Robert Staka for his 

critique and suggestions for improving ah earlier version of Table 3, 

4_ 

Based on discussion notes supplied to the author by Joseph Hansen* 
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